feat(data): record which feed a cached series came from (#696, part 1… - #710
Merged
Conversation
… of 2) Schema v17. The provenance half of #696, landing BEFORE the gate that reads it -- a gate built on a feed inferred at read time would encode the bug it exists to fix. THE BUG. keel's liquidity statistic is `median(volume * close)` over cached candles, and every threshold keyed on it was calibrated against venue-reported volume. That is coherent on a crypto exchange: Coinbase's own volume IS the scale the floor was chosen against. It is not coherent on Alpaca's IEX feed, which reports one US equity exchange's own executions. The cost-fidelity run measured MSFT -- one of the most liquid securities in the world -- cached at $186M/day against a model anchored at $500M, and priced it as a thin asset. Nothing recorded which feed produced those bars, so the answer was whatever `broker.data_feed` happened to be loaded when someone later read the series. WHAT LANDS HERE. `candle_series_feed`, keyed on `(product_id, granularity, feed)` -- NOT a mutable column on `candles`. A series fetched under both feeds records BOTH rows, because that is what happened; a single overwritable column would let the most recent fetch erase the fact that most of the bars came from somewhere narrower, and a mixed series is exactly the case a reader most needs warning about. NO BACKFILL, and that is the point rather than a shortcut. Seeding rows from the current `broker.data_feed` would manufacture precisely the claim the table exists to make checkable, for years of bars that may have come from elsewhere. An empty table means "unrecorded", which is TRUE of every pre-existing series and is a different statement from "consolidated" -- `reports_consolidated_volume` returns `None` for the first and `False`/`True` for the second, and callers must not conflate the absence of evidence with evidence. `keel/data/feed_scope.py` carries the design, whose core is an ASYMMETRY: a single-venue feed's volume is a LOWER BOUND on consolidated volume, never an upper one. Volume at or above the floor on a partial feed is therefore a CONCLUSIVE pass -- if a name traded that much on one venue alone it necessarily traded at least that much in total -- while volume below the floor licenses no claim either way and must be refused as unmeasured rather than reported as thin. That is what lets this be honest without encoding any venue's market share: the bound holds for any share below 100%, so nothing here has a percentage in it that market structure could invalidate. `test_the_bound_needs_no_market_share` pins that no figure leaks into executable code, because the one thing that must never happen is someone scaling a volume statistic by an assumed share. An unrecognised feed id reads as PARTIAL, which costs a conclusive pass on a consolidated feed nobody declared and never grants one on a narrow feed nobody declared. Adapters declare their own feed (`volume_feed_id`), qualified by venue -- `alpaca:iex` / `alpaca:sip`, `coinbase` -- never bare `iex`, which would collide the day another venue routes there. Declaration is structural and optional: an adapter that declares nothing records nothing, so unrecorded stays unrecorded rather than every crypto series silently acquiring a provenance nobody established. Forcing all five adapters to implement it would invite a placeholder, which is the one value that must never enter this table. Tests: 31 across three files. tests/data/test_series_feed_provenance.py -- 16, the storage contract tests/data/test_feed_scope.py -- 11, the scope verdict tests/data/test_feed_provenance_is_wired.py -- 4, the fetch path actually carries the declaration; without these the whole thing degrades silently to pre-issue behaviour with a schema and a module suggesting otherwise The provenance and wiring suites were written test-first. `feed_scope.py` was NOT -- I wrote it before its tests, which is the wrong order; the 8-mutant run below is what establishes those tests constrain the code rather than merely describe it. Mutation-verified, 8 mutants, all killed: unrecorded reading as consolidated; `any` for `all` on a mixed series; `alpaca:iex` declared consolidated; a blank declaration accepted; provenance recorded for an empty batch; an empty feed string accepted; a re-fetch overwriting `first_seen_ts`; a missing table raising instead of reading empty. Five existing version pins move 16 -> 17. They are literals on purpose -- a deliberate speed bump that makes a schema bump a decision someone acknowledged. Refs #696 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NzuKAe2RVrPt9acVAWjRyL
…ch` (#696) Four findings from the review of #710, fixed on the branch. The first one meant the feature was very nearly inert. 1. THE ONE THAT MATTERED. Only `history.ensure_history` carried the feed declaration. `market_feed.poll_once` -- the path `agent.run_once` uses on EVERY cycle in every deployment -- and `repair.repair_series` both dropped it, so in a real database almost every bar would have been written with no provenance and `candle_series_feed` would have stayed nearly empty while a schema, a module and 31 tests said otherwise. `keel fetch` is not how bars normally arrive. The wiring tests did not catch it because they exercised `ensure_history` alone: they proved the mechanism worked, never that every writer used it. Both paths now resolve the feed once and thread it down, and there are tests per WRITER rather than per mechanism. `backfill` resolves its own rather than relying on one being threaded in -- it has no in-tree caller today, and a writer that silently records nothing is precisely the failure this table exists to prevent. 2. `get_series_feeds` / `get_series_feed_window` rescued `sqlite3.OperationalError` wholesale, which is also what a LOCK TIMEOUT raises -- and keel reads and writes this file from more than one process by design. A lock would have been reported as "scope unrecorded" for a series whose scope is on disk: exactly the `None`-vs-`False` conflation `feed_scope` exists to prevent, silently. Now only `no such table` is rescued; anything else propagates. 3. `DeclaresVolumeFeed` was declared, exported and never used, because `volume_feed_of` reads the attribute with `getattr` (presence is not enough -- the value must also be a non-blank string). A Protocol nothing checks drifts from the check that matters, so a test now asserts both adapters satisfy it. 4. Three closing parens sat at column 0 from a scripted edit. `ruff check` passes on them because `ruff format` is not a CI gate, which is why they survived. Mutation-verified, 4 mutants. One SURVIVED on the first pass: reverting the narrowed rescue in (2) to a bare `return ()` broke nothing, because I had fixed the catch without writing a test that a lock must propagate. `test_a_locked_database_is_raised_not_read_as_unrecorded` closes that, and all four now die. Refs #696 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NzuKAe2RVrPt9acVAWjRyL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
… of 2)
Schema v17. The provenance half of #696, landing BEFORE the gate that reads it -- a gate built on a feed inferred at read time would encode the bug it exists to fix.
THE BUG. keel's liquidity statistic is
median(volume * close)over cached candles, and every threshold keyed on it was calibrated against venue-reported volume. That is coherent on a crypto exchange: Coinbase's own volume IS the scale the floor was chosen against. It is not coherent on Alpaca's IEX feed, which reports one US equity exchange's own executions. The cost-fidelity run measured MSFT -- one of the most liquid securities in the world -- cached at $186M/day against a model anchored at $500M, and priced it as a thin asset. Nothing recorded which feed produced those bars, so the answer was whateverbroker.data_feedhappened to be loaded when someone later read the series.WHAT LANDS HERE.
candle_series_feed, keyed on(product_id, granularity, feed)-- NOT a mutable column oncandles. A series fetched under both feeds records BOTH rows, because that is what happened; a single overwritable column would let the most recent fetch erase the fact that most of the bars came from somewhere narrower, and a mixed series is exactly the case a reader most needs warning about.NO BACKFILL, and that is the point rather than a shortcut. Seeding rows from the current
broker.data_feedwould manufacture precisely the claim the table exists to make checkable, for years of bars that may have come from elsewhere. An empty table means "unrecorded", which is TRUE of every pre-existing series and is a different statement from "consolidated" --reports_consolidated_volumereturnsNonefor the first andFalse/Truefor the second, and callers must not conflate the absence of evidence with evidence.keel/data/feed_scope.pycarries the design, whose core is an ASYMMETRY: a single-venue feed's volume is a LOWER BOUND on consolidated volume, never an upper one. Volume at or above the floor on a partial feed is therefore a CONCLUSIVE pass -- if a name traded that much on one venue alone it necessarily traded at least that much in total -- while volume below the floor licenses no claim either way and must be refused as unmeasured rather than reported as thin. That is what lets this be honest without encoding any venue's market share: the bound holds for any share below 100%, so nothing here has a percentage in it that market structure could invalidate.test_the_bound_needs_no_market_sharepins that no figure leaks into executable code, because the one thing that must never happen is someone scaling a volume statistic by an assumed share.An unrecognised feed id reads as PARTIAL, which costs a conclusive pass on a consolidated feed nobody declared and never grants one on a narrow feed nobody declared.
Adapters declare their own feed (
volume_feed_id), qualified by venue --alpaca:iex/alpaca:sip,coinbase-- never bareiex, which would collide the day another venue routes there. Declaration is structural and optional: an adapter that declares nothing records nothing, so unrecorded stays unrecorded rather than every crypto series silently acquiring a provenance nobody established. Forcing all five adapters to implement it would invite a placeholder, which is the one value that must never enter this table.Tests: 31 across three files.
tests/data/test_series_feed_provenance.py -- 16, the storage contract
tests/data/test_feed_scope.py -- 11, the scope verdict
tests/data/test_feed_provenance_is_wired.py -- 4, the fetch path actually
carries the declaration; without these the whole thing degrades silently to
pre-issue behaviour with a schema and a module suggesting otherwise
The provenance and wiring suites were written test-first.
feed_scope.pywas NOT -- I wrote it before its tests, which is the wrong order; the 8-mutant run below is what establishes those tests constrain the code rather than merely describe it.Mutation-verified, 8 mutants, all killed: unrecorded reading as consolidated;
anyforallon a mixed series;alpaca:iexdeclared consolidated; a blank declaration accepted; provenance recorded for an empty batch; an empty feed string accepted; a re-fetch overwritingfirst_seen_ts; a missing table raising instead of reading empty.Five existing version pins move 16 -> 17. They are literals on purpose -- a deliberate speed bump that makes a schema bump a decision someone acknowledged.
Refs #696
Claude-Session: https://claude.ai/code/session_01NzuKAe2RVrPt9acVAWjRyL
What & why
Tests-first evidence
Gates (all must pass)
uv run ruff checkcleanuv run mypycleanuv run pytest -qgreenScope check
leave checked only if true, and if so: cite the source and open the discussion
BEFORE review (CONTRIBUTING.md, "Governance: rulings vs. machinery").